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DETAILED ACTION 

In view of the Appeal brief filed on April 25, 2008, PROSECUTION IS HEREBY 
REOPENED. The new grounds of rejection are set forth below. 

To avoid abandonment of the application, appellant must exercise one of the 
following two options: 

(1 ) file a reply under 37 CFR 1.111 (if this Office action is non-final) or a reply 
under 37 CFR 1.113 (if this Office action is final); or, 

(2) initiate a new appeal by filing a notice of appeal under 37 CFR 41 .31 followed 
by an appeal brief under 37 CFR 41 .37. The previously paid notice of appeal fee and 
appeal brief fee can be applied to the new appeal. If, however, the appeal fees set forth 
in 37 CFR 41 .20 have been increased since they were previously paid, then appellant 
must pay the difference between the increased fees and the amount previously paid. 

A Supervisory Patent Examiner (SPE) has approved of reopening prosecution by 
signing below: 

/Richemond Dorvil/ 

Supervisory Patent Examiner, Art Unit 2626. 

Response to Arguments 

Applicant's arguments, see Appeal Brief, filed April 25, 2008, with respect to the 
35 U.S.C. §112 2 nd rejection of claim 8, 9, 19, 21 and 22 have been fully considered and 
are persuasive. The rejections are withdrawn. 
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Applicant's arguments, see Remarks, filed April 25, 2008, with respect to the 
rejection(s) of claim(s) 1-17 and 19-22 under 35 U.S.C. §103 have been fully 
considered and are persuasive. Therefore, the rejection has been withdrawn. 
However, upon further consideration, a new ground(s) of rejection is made in view of 
Johnson. 

Claim Rejections - 35 USC § 103 

The following is a quotation of 35 U.S.C. 1 03(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

Claims 1-4, 6-12, 14-22 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Johnson (9,748,974) in view of W3C ("Natural Language Semantics 
Markup Language for the Speech Interface Framework" 2000). 

As per claim 1 , Johnson discloses a method of input conversion to output, said 
conversion method comprising: 

entering a first stage input and a second input (column 3 lines 29 - 60, the 
system enables multimodal input, for example speech and keyboard input); 

translating the first input and the second input into an electronic format (column 3 
lines 63-67, the speech and non-speech input are processed by a natural language 
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processor as part of a computer system, therefore the inputs must have been translated 
into electronic format); 

Johnson does not disclose converting the electronic format into standard XML 
encoded with accessibility information, transforming the standard XML encoded with 
accessibility information into individual version of XML dependent on desired output and 
utilizing a rendering engine to modify the individual version of XML into a format for the 
output. However, Johnson does disclose determining verbal and non-verbal semantic 
information (accessibility information) from the input and converting that into an 
individual version dependent on the desire output (column 3 line 63 - column 5 line-32). 
The combined multimodal input, including a spoken phrase and a typed input or item-in- 
focus input, is sent to a natural language processor where it is processed by a 
parser/semantic interpreter and then a dialog manager. The output of the natural 
language processor is then sent to a dispatcher, which determines which application 
should process the information. Once a determination has been made the information is 
sent to the corresponding application where a an appropriate response is generated 
(rendering engine), for example as synthesized speech, text pop up window or video 
clip, depending on the nature of the request and the current application. In addition, 
XML is a well known markup language which is similar to HTML but was designed to 
structure, store and carry data rather than display data. XML was designed to be self- 
descriptive: therefore tags are not predefined and must be defined by the user. 
Elements and attributes- which provide additional information about the elements- are 
defined by the user and can be extended to carry additional information. The W3C 
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international consortium has also published specifications for an XML language that 
provides semantic interpretations for a variety of speech and language inputs (W3C). 
The Natural Language Semantics Markup supports a Voice Browser, which functions 
with a variety of input modalities and can represent meta-dialog information or meta- 
task utterances (section 1 .1 and section 2.8). XML files can also be converted into a 
different XML file using XSLT, or Extensible Stylesheet Language Transformations. 
XSLT is a language used to convert one element on an XML document into an element 
of another XML document. 

Therefore it would have been obvious to one of ordinary skill in the art at the time 
of the invention to convert the electronic format into standard XML encoded with 
accessibility information, transform the standard XML encoded with accessibility 
information into individual version of XML dependent on desired output and utilize a 
rendering engine to modify the individual version of XML into a format for the output in 
Johnson, since one of ordinary skill has good reason to pursue the options within his or 
her technical grasp. Additionally, it would have been obvious to combine the known 
multimodal interface in Johnson with XML, since XML data is stored as plain text data, 
and can therefore be easily shared amongst different applications and exchanged 
between incompatible systems. 

As per claim 2, Johnson in view of W3C disclose the conversion method of claim 1 , 
and Johnson further discloses wherein said entering step includes speaking (column 3 
lines 44-46. 
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.As per claim 3, Johnson in view of W3C disclose the conversion method of claim 1 , 
neither disclose wherein said entering step includes gesturing. However, Johnson does 
disclose that the input to the system can include any modality (column 3 lines 44-46). 
This suggests the system is capable of interpreting gestures as in input. 

Therefore it would have been obvious to one of ordinary skill in the art at the time 
of the invention to have an entering step that includes gesturing in Johnson and WC3, 
since it would enable users with speech disabilities to easily enter data. 

As per claim 4, Johnson in view of W3C disclose the conversion method of claim 1 , 
and Johnson further discloses wherein said entering step includes writing (column 3 
lines 44-46). 

As per claim 6, Johnson in view of W3C disclose the conversion method of claim 1 , 
and Johnson further discloses wherein said entering step includes semantic 
information (column 3 lines 49-54). 

As per claim 7, Johnson in view of W3C disclose the conversion method of claim 1 , 
however neither disclose wherein said entering step includes providing formatting and 
structure for text. However, Johnson does disclose that a response from the system 
includes text which is pasted into the current application (column 4 lines 49-53). 
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Therefore it would have been obvious to one of ordinary skill in the art at the time 
of the invention to have an entering step that includes formatting and structure for text in 
Johnson and WC3, since it would enable the user to adjust the output text according to 
their desired viewing preferences. 

As per claim 8, Johnson in view of W3C disclose the conversion method of claim 2, 
and Johnson further discloses wherein said translating step includes the use of 
automatic speech recognition of the first input and the second input to produce the 
electronic format, the first input including verbal information, and the second input 
including non-verbal information (column 3 lines 44-49). 

As per claim 9, Johnson in view of W3C disclose the conversion method of claim 3, 
however neither disclose wherein said translating step includes the use of sign 
language recognition of the first input and the second input to produce the electronic 
format, the first input including verbal information, and the second input including non- 
verbal information. However, Johnson does disclose that the input to the system can 
include any modality (column 3 lines 44-46). This suggests the system is capable of 
interpreting gestures, including sign language, as in input. 

Therefore it would have been obvious to one of ordinary skill in the art at the time 
of the invention to have a translating step that includes the use of sign language 
recognition in Johnson and WC3, since it would enable users with disabilities to easily 
enter data. 
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As per claims 10 and 1 1 , Johnson in view of W3C disclose the conversion method of 
claim 1, but neither disclose wherein the individual version of XML is XHTML or 
VoiceXML for desired output. However, Johnson does disclose producing an output 
that is appropriate based on the request and current application (column 4 lines 49-53). 
In addition, XML files can easily be converted into a different type of XML file using 
XSLT, or Extensible Stylesheet Language Transformations. 

Therefore it would have been obvious to one of ordinary skill in the time of the 
invention to have the individual version of XML as XHTML or VoiceXML in Johnson 
and WC3, since it would enable the user to adjust the output text according to their 
desired viewing preferences. 

As per claim 12, Johnson in view of W3C disclose the conversion method of claim 1 , 
and Johnson further discloses wherein the accessibility information includes non-verbal 
information (column 3 line 63 - column 5 line-32). 

As per claim 14, Johnson in view of W3C disclose the conversion method of claim 1 1 , 
and Johnson further discloses wherein the desired output is synthesized speech 
(column 7 lines 53-55). 

As per claims 15, 16 and 17, Johnson in view of W3C disclose the conversion method 
of claim 1 2, but neither discloses wherein the desired output is virtual sign language, 
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electronic Braille, or electronic large print. However, Johnson does disclose that the 
output is an appropriate response based on the request and current application, and 
includes synthesized speech, text, a video clip or the like (column 4 lines 49-51). This 
suggests that the system is capable of producing virtual sign language, electronic Braille 
or electronic large print as an output. 

Therefore it would have been obvious to one of ordinary skill in the art at the time 
of the invention to output the information as virtual sign language, electronic Braille or 
electronic large print in Johnson and WC3, since it would enable users with disabilities 
to adjust the output to a format that that they can easily interpret. 

As per claim 18, Johnson in view of W3C disclose the conversion method of claim 1 , 
but neither disclose wherein said transforming step includes the use of extensible 
stylesheet language transformations to transform the standard XML encoded with 
accessibility information into the individual version of XML dependent on desired output. 
However, Extensible Stylesheet Language Transformations are known and commonly 
used to convert one element on an XML document into an element of another XML 
document. 

Therefore it would have been obvious to one of ordinary skill in the art at the time 
of the invention to use extensible stylesheet language transformations to transform the 
standard XML encoded with accessibility information into the individual version of XML 
dependent on desired output in Johnson and WC3, since one of ordinary skill in the art 
has good reason to pursue the options within his or her technical grasp in order to 
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achieve the predictable result of reliably and accurately transforming an XML file into a 
different XML file. 

As per claim 19, Johnson discloses a method of input conversion to output, said 
conversion method comprising: 

entering a first input including verbal information and a second input including 
non- verbal information, the second input being entered in parallel with the first input, 
the second input being entered via a same medium as the first input, the same medium 
being one of a visual medium and an audial medium (column 3 line 63 - column 5 line- 
32, the system accepts multimodal input that is either typed or handwritten along with 
non-speech input that is entered through mouse or touch screen); 

translating the first input and the second input into an electronic format (column 3 
lines 63-67, the inputs are processed by a natural language processor as part of a 
computer system, therefore the inputs must have been translated into electronic 
format); 

Johnson does not disclose converting the electronic format into standard XML 
encoded with accessibility information, transforming the standard XML encoded with 
accessibility information into individual version of XML dependent on desired output and 
utilizing a rendering engine to modify the individual version of XML into a format for the 
output. However, Johnson does disclose determining verbal and non-verbal semantic 
information (accessibility information) from the input and converting that into an 
individual version dependent on the desire output (column 3 line 63 - column 5 line-32). 
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The combined multimodal input, including a spoken phrase and a typed input or item-in- 
focus input, is sent to a natural language processor where it is processed by a 
parser/semantic interpreter and then a dialog manager. The output of the natural 
language processor is then sent to a dispatcher, which determines which application 
should process the information. Once a determination has been made the information is 
sent to the corresponding application where a an appropriate response is generated 
(rendering engine), for example as synthesized speech, text pop up window or video 
clip, depending on the nature of the request and the current application. In addition, 
XML is a well known markup language which is similar to HTML but was designed to 
structure, store and carry data rather than display data. XML was designed to be self- 
descriptive: therefore tags are not predefined and must be defined by the user. 
Elements and attributes- which provide additional information about the elements- are 
defined by the user and can be extended to carry additional information. The W3C 
international consortium has also published specifications for an XML language that 
provides semantic interpretations for a variety of speech and language inputs {W3C). 
The Natural Language Semantics Markup supports a Voice Browser, which functions 
with a variety of input modalities and can represent meta-dialog information or meta- 
task utterances (section 1 .1 and section 2.8). XML files can also be converted into a 
different XML file using XSLT, or Extensible Stylesheet Language Transformations. 
XSLT is a language used to convert one element on an XML document into an element 
of another XML document. 
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Therefore it would have been obvious to one of ordinary skill in the art at the time 
of the invention to convert the electronic format into standard XML encoded with 
accessibility information, transform the standard XML encoded with accessibility 
information into individual version of XML dependent on desired output and utilize a 
rendering engine to modify the individual version of XML into a format for the output in 
Johnson, since one of ordinary skill has good reason to pursue the options within his or 
her technical grasp. Additionally, it would have been obvious to combine the known 
multimodal interface in Johnson with XML, since XML data is stored as plain text data, 
and can therefore be easily shared amongst different applications and exchanged 
between incompatible systems. 

As per claim 20, Johnson discloses a method of input conversion to output, said 
conversion method comprising: 

entering a first input including text information and a second input including 
semantic information (column 3 line 63 - column 5 line-32, the system accepts 
multimodal input that is either typed or handwritten (text information) along with non- 
speech input that is entered through mouse or touch screen, such as item-in-focus data 
(semantic information)); 

translating the first input and the second input into an electronic format (column 3 
lines 63-67, the inputs are processed by a natural language processor as part of a 
computer system, therefore the inputs must have been translated into electronic 
format); 
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Johnson does not disclose converting the electronic format into standard XML 
encoded with accessibility information, transforming the standard XML encoded with 
accessibility information into individual version of XML dependent on desired output and 
utilizing a rendering engine to modify the individual version of XML into a format for the 
output. However, Johnson does disclose determining verbal and non-verbal semantic 
information (accessibility information) from the input and converting that into an 
individual version dependent on the desire output (column 3 line 63 - column 5 line-32). 
The combined multimodal input, including a spoken phrase and a typed input or item-in- 
focus input, is sent to a natural language processor where it is processed by a 
parser/semantic interpreter and then a dialog manager. The output of the natural 
language processor is then sent to a dispatcher, which determines which application 
should process the information. Once a determination has been made the information is 
sent to the corresponding application where a an appropriate response is generated 
(rendering engine), for example as synthesized speech, text pop up window or video 
clip, depending on the nature of the request and the current application. In addition, 
XML is a well known markup language which is similar to HTML but was designed to 
structure, store and carry data rather than display data. XML was designed to be self- 
descriptive: therefore tags are not predefined and must be defined by the user. 
Elements and attributes- which provide additional information about the elements- are 
defined by the user and can be extended to carry additional information. The W3C 
international consortium has also published specifications for an XML language that 
provides semantic interpretations for a variety of speech and language inputs (W3C). 
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The Natural Language Semantics Markup supports a Voice Browser, which functions 
with a variety of input modalities and can represent meta-dialog information or meta- 
task utterances (section 1 .1 and section 2.8). XML files can also be converted into a 
different XML file using XSLT, or Extensible Stylesheet Language Transformations. 
XSLT is a language used to convert one element on an XML document into an element 
of another XML document. 

Therefore it would have been obvious to one of ordinary skill in the art at the time 
of the invention to convert the electronic format into standard XML encoded with 
accessibility information, transform the standard XML encoded with accessibility 
information into individual version of XML dependent on desired output and utilize a 
rendering engine to modify the individual version of XML into a format for the output in 
Johnson, since one of ordinary skill has good reason to pursue the options within his or 
her technical grasp. Additionally, it would have been obvious to combine the known 
multimodal interface in Johnson with XML, since XML data is stored as plain text data, 
and can therefore be easily shared amongst different applications and exchanged 
between incompatible systems. 

As per claim 21 , Johnson in view of W3C disclose the conversion method of claim 19 
wherein the same medium comprises automatic speech recognition (column 3 lines 44- 
49). 
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As per claim 21 , Johnson in view of W3C disclose the conversion method of claim 1 9, 
however neither disclose wherein the medium comprises sign language recognition. 
However, Johnson does disclose that the input to the system can include any modality 
(column 3 lines 44-46). This suggests the system is capable of interpreting gestures, 
including sign language, as in input. 

Therefore it would have been obvious to one of ordinary skill in the art at the time 
of the invention to have a translating step that includes the use of sign language 
recognition in Johnson and WC3, since it would enable users with disabilities to easily 
enter data. 



Conclusion 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Dorothy Sarah Siedler whose telephone number is 571- 
270-1067. The examiner can normally be reached on Mon-Thur 9:30am-5:30pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Richemond Dorvil can be reached on 571-272-7602. The fax phone 
number for the organization where this application or proceeding is assigned is 571- 
273-8300. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 

DSS 

/Richemond Dorvil/ 

Supervisory Patent Examiner, Art Unit 2626 



